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Background / Context: 

Description of prior research and its intellectual context. 


When presenting results from rigorous, field-based evaluations of educational interventions, 
researchers often reveal methodological barriers they face in designing studies to assessing 
program effectiveness. Methodologists seek out these conference sessions to gain insights into 
these barriers so they can derive new methodological approaches to provide sorely needed 
solutions that allow researchers to generate evidence that helps solve the most relevant and 
pressing problems in education reform. The evaluations funded by the Investing in Innovation 
Fund (i3) present a timely opportunity for identifying relevant and pressing issues arising in 
current education evaluations. With over $1 billion dollars invested in the implementation and 
evaluation of 1 17 education interventions, the i3 program represents a major investment by the 
U.S. Department of Education in educational interventions that will both serve students, teachers 
and schools, and generate evidence about intervention effectiveness. Each grant supports both 
an impact and an implementation evaluation, and span topics from school readiness to college 
transition and all points in between. The i3 program has funded interventions at various stages, 
from development grants looking to implement and evaluate new programs, to validation grants 
that investigate the effectiveness of interventions already shown to be effective, to scale-up 
grants that aim to bring proven approaches to a national scale. The breadth and number of i3 
evaluations provides a unique opportunity to leam about the features of, and challenges 
encountered in, the design and implementation of the evaluations. Looking across these 
evaluations will guide methodologists who endeavor to solve practical problems encountered in 
field-based evaluations, focusing our collective energy on those that are most pressing. 

Researchers from the National Evaluation of i3 are in a unique position to describe and 
summarize the barriers to high quality research the i3 evaluators face. The NEi3 is tasked with 
providing one-on-one technical assistance to the i3 evaluators, aimed at identifying and 
mitigating problems implementing rigorous evaluations. Through this work, we are able to 
identify the areas in which new methodological approaches would be most helpful in raising the 
quality of evidence generated by education evaluations. 


Purpose / Objective / Research Question / Focus of Study: 

Description of the focus of the research. 


This paper will answer the following research questions: 

1. What designs are being used to evaluate the i3 interventions? 

2. How has the choice of design varied by: Grant type (development, validation, and scale- 
up); intervention target ages/grades (e.g., pre-school interventions, elementary, middle, 
high school, transition to college); intervention target outcome domains (e.g., ELA, math, 
science, arts achievement, socio-emotional; behavior regulation; GPA, credits, 
attendance, course taking, FAFSA completion)? 
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3. What are the threats to causal inference that most commonly arise in the design and 
implementation of the i3 evaluations? 

Setting: 

Description of the research location. 

The i3 evaluations span all regions of the U.S. and include interventions that are being 
implemented in urban, suburban and rural settings. 

Population / Participants / Subjects: 

Description of the participants in the study: who, how many, key features, or characteristics. 

The i3 evaluations span all age/grade levels from pre-K through college transition and include 
interventions targeted to a range of populations spanning from high minority urban settings to 
rural, low minority settings. 


Intervention / Program / Practice: 

Description of the intervention, program, or practice, including details of administration and duration. 

All i3 grants must include an impact and an implementation evaluation. Scale-up and validation 
grants are expected to include impact designs that would meet What Works Clearinghouse 
(WWC) evidence standards with or without reservations. Development grants are encouraged to 
implement impact designs that would meet WWC standards with or without reservations, but it 
is expected that some will have designs that support only evidence of “promise” (e.g., a pre-post 
design or interrupted time series without comparison group). Most i3 grants are for five years 
and most have three- or four-year evaluation periods. 

Research Design: 

Description of the research design. 


This will be a descriptive study of the research designs used in the i3 grant program. The i3- 
funded evaluations are using cluster randomized designs, randomization of individuals, 
regression discontinuity designs, quasi-experimental designs (QEDs) that establish baseline 
equivalence of clusters (schools), QEDs that establish baseline equivalence of individuals, short 
interrupted time series designs with and without comparison groups, value-added approaches, 
synthetic comparison group approaches, off-year comparison groups, and pre-post only 
assessments. We will summarize the number of evaluations using each design, and cross- 
tabulate the designs with the categories listed above (Grant type (development, validation, and 
scale-up); intervention target ages/grades (e.g., pre-school interventions, elementary, middle, 
high school, transition to college); intervention target outcome domains (e.g., ELA, math, 
science, arts achievement, socio-emotional; behavior regulation; GPA, credits, attendance, 
course taking, FAFSA completion). 

We will also summarize the issues encountered by the evaluations, focusing on those that 
threaten the internal validity of the estimates of the effect of the intervention, including low 
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power, treatment group cross-over, poorly defined or measured counterfactual, n=l confounds, 
comparison groups comprised of units that were offered but refused treatment, poor alignment 
between intervention target and available outcome measures, and mid-study changes to state 
assessments. 

Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 


Abt Associates Inc. is the lead contractor for the National Evaluation of the Investing in 
Innovation Fund (NEi3). As part of this contract, we provide evaluation technical assistance to 
all of the evaluations of all of the i3 grants and are therefore intimately familiar with the designs 
being used. The primary data for this study will come from a systematic review and abstraction 
of the design and analysis plans from all 1 17 evaluations. Results will be tabulated by study 
design type, grant type, intervention target ages/grades, and intervention target outcome domain. 
Additional data regarding themes of threats to causal inferences and design and implementation 
challenges will come from structured interviews of all 21 technical assistance providers on the 
contract. 

Findings / Results: 

Description of the main findings with specific details. 


Findings will be forthcoming. 

Conclusions: 

Description of conclusions, recommendations, and limitations based on findings. 


The results of this study will provide methodologists with information on the prevalence of 
various designs being implemented in a high profile grant program, and the methodological 
challenges encountered in those designs. This information will help methodologists form 
priorities of where to focus their work on methodological innovations. 
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Appendices 

Not included in page count. 


Appendix A. References 

References are to be in APA version 6 format. 
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Appendix B. Tables and Figures 
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